fix(algorithms): raise instead of asserting on the skipped-reference-KL pairing - #3854

…KL pairing

Sync PPO and sync GRPO both guard skip_reference_policy_logprobs_calculation
against a non-zero reference_policy_kl_penalty with a bare, message-less

    assert master_config.loss_fn.reference_policy_kl_penalty == 0

python -O removes it entirely, so the run proceeds to train against a KL term
whose reference logprobs were never computed. Even with asserts enabled, a
message-less one says nothing about which two settings conflict.

async_ppo_train already does this as an if + ValueError. This is the same
defect class yuki-97 raised on NVIDIA-NeMo#3262 ("assert backend == 'vllm' gets stripped
under python -O"), which was converted there but left in these two siblings.

Both now use the async form, verbatim, so the three read alike.

Tests run under python -O as well as normally, which is what separates this
from a cosmetic change: with the assert restored, both modes fail.

Signed-off-by: Tianyi Zhang <zhangtianyi975@gmail.com>
Signed-off-by: Tianyi Zhang <123608656+tianyi-zhang-02@users.noreply.github.com>

…ed one

grpo_train_sync's assert is unreachable in practice: grpo.setup checks the
same pairing before the train loop starts, so on GRPO that is the one a user
hits. It was an assert too, so under python -O both were stripped and nothing
was left -- converting only the train-loop copy would have fixed the copy
nobody reaches.

PPO is the other way round: there is no setup-time equivalent, so the one in
ppo_train is the only guard and converting it was already right.

Keeps the grpo_sync conversion as well. It costs nothing and the three sites
now read alike, which was the point.

Test asserts on grpo.setup's source rather than calling it, since calling it
needs a cluster. What it pins is that the reachable guard is not an assert.

Signed-off-by: Tianyi Zhang <123608656+tianyi-zhang-02@users.noreply.github.com>

Signed-off-by: Tianyi Zhang <123608656+tianyi-zhang-02@users.noreply.github.com>